708763 Observatory Middleware Framework : OMF Project Space

Abstract and Proposal

Proposal

Goals of the Project

1. Federate data streams from multiple observatories
2. Create common instrument access and management
3. Provide fundamental mechanism to enforce policies
4. Captures and forwards rich metadata including data provenance
5. Provide mechanism to handle data and events in a uniform way
6. Ability to readily integrate and configure instruments, data streams, processing and storage resources with indifference to resource implementations
7. Uniform interface to observatory resources
8. Service integration framework

Approach

1. Integrate existing sensors, services, and grid functionality using an Enterprise Service Bus (ESB), specifically Mule.
2. Three functional areas where this integration occurs:
   1. Instrument Access and Management
   2. Stream Processing
   3. Services (Specifically Governance and Provenance)
3. Implementation will be validated using bench tests and through pilot implementations on MOOS and MARS
4. Will use the integration of SIAM, ROADNet, and direct instrument as an example
5. Observer can view and subscribe to data streams
6. Provide a comprehensive suite of functional capabilities integrated across a complete range of installed systems (including simulated systems).
7. We will provide benchmarks
8. Develop specification and implementation of standard instrumentation interfaces between the Sensor Enterprise Service Bus and the Instrument Proxy, including the complete semantics of the message exchange between those services
9. Develop specification of a set of "standard instrument" capabilities that an Instrument Proxy will represent and that can be mapped to the functionality of either specific instruments or observatory representations of its instruments
10. Develop specification of the stream protocols that will carry data, and stream-characterizing metadata, for use by applications
11. Provide evaluation and adoption of a suitable content standard for description of data sources and data provenance, enabling plug-and-play functionality and rich metadata integration of data streams
12. Adopt suitable vocabularies for use within the content standards described above
13. A generic Instrument Proxy that bridges the SESB protocols to a set of typical instrument capabilities
14. Observatory-specific Instrument Proxies that bridge SESB protocols to instrument protocols as presented by existing observatories like MOOS (via SIAM) and US Array (via ROADNet)
15. Modification of the instrument protocols presented by existing observatory infrastructures (like SIAM and ROADNet) to provide the capabilities expected by observatory-specific Instrument Proxies.
16. Governance will be done by controlling the flow of messages and data streams and will be done using:
    1. Internet2 Signet work
    2. Sun XACML implementation
17. Authentication and identity management will be done with integration of technologies from:
    1. Grid Security Infrastructure
    2. Sibboleth (using Kerberos/LDAP)
    3. GridShib (using Kerberos/LDAP)
    4. GridGrouper
    5. JAAS (for Mule ESB)
18. For client tools, we will look at using:
    1. Nagios for hosts and network services
    2. Ganglia for monitoring high-performance computing systems
    3. StreamBase stream processing engine
    4. Coral8 Engine for event detection and processing

Requirements

1. Correlate data readings from multiple sensors
2. Establish relationships between the various correlated data
3. Scientist can discover data despite naming differences
4. Immediate access to real-time data streams from different observatories
5. Stream data processing
6. Respond to events by:
   1. increasing or altering data acquisition rates
   2. taking operational actions to ensure robust data collection
   3. allocating additional sensors to take extra observations to characterize the anomaly
7. Promote collaboration among researchers
8. Provide common mechanism for managing data streams
9. Provide common mechanism for automating the creation of derived data from data streams
10. Provide mechanisms for integration of data and models
11. Provide support for workflow automation
12. Create and operate virtual observatory

Architecture

MBARI Statement of Work

This statement of Work defines the effort required to demonstrate instrument federation using an ESB. MBARI's work is to

1. build instrument control and data access procedures as part of the MBARI SIAM (Software infrastructure and application for MOOS) infrastructure implementing agreed to standard protocols
2. implement the necessary protocols in low level hardware that resides with individual instruments
3. implement the documented reference implementation on ROADNet/Antelope.

Detailed tasks

1. Build up InstrumentProxyStrategy commands for instrument control (e.g. turn on/off, etc.).
2. Augment the InstrumentProxy classes to handle the instrument control commands.
3. Perform the SIAM and ROADnet integration to handle those commands.
4. Document the reference implementation.
5. Develop software to run InstrumentProxy on Digi Connect to demonstrate plugging a device directly onto the ESB.

InGrid Demonstration

One target for this project was to develop a demonstration that could be presented at the InGrid conference. Here is a logical

Related Resources

1. OMF Wiki
2. Meeting Notes
3. Demo Plans
4. Post-Demo Thoughts
5. OMF Mule Architecture
6. OMF Project Kickoff

Development Notes

1. SVN URL (you need an account and permissions): https://svn.ncsa.uiuc.edu/svn/omf
2. Build And Run SENSORS CTD

MRG Project

As a side project for the OOI CyberInfrastructure Team, we worked on moving the code for utilizing the ESB to the QPID messaging framework that is running under RedHat's MRG software. You can find that work documented here:

MRG Development

mule-config.xml (text/xml)
Demo Story.jpg (image/jpeg)